You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 23:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Nov 18, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991160,6rtkit-daemon0-21swapper/312:02:252
1623991110,11rtkit-daemon1753-21gnome-shell07:23:422
1623991110,10rtkit-daemon1753-21gnome-shell08:59:472
1623991040,5rtkit-daemon0-21swapper/307:40:012
1623991030,5rtkit-daemon0-21swapper/308:32:082
1623991020,6rtkit-daemon0-21swapper/307:59:492
1623991020,5rtkit-daemon0-21swapper/307:34:032
1623991010,6rtkit-daemon0-21swapper/311:08:472
1623991010,5rtkit-daemon0-21swapper/308:42:262
1623991000,7rtkit-daemon1387-21kerneloops11:50:312
162399960,7rtkit-daemon2197014-21PK-Backend09:06:142
162399880,71rtkit-daemon0-21swapper/207:07:142
8218828469,9sleep00-21swapper/007:05:540
1871952830,4sleep20-21swapper/212:05:172
8244428262,13sleep30-21swapper/307:09:153
162399810,6rtkit-daemon0-21swapper/307:28:213
162399800,6rtkit-daemon0-21swapper/311:05:003
162399790,6rtkit-daemon0-21swapper/311:25:223
162399790,5rtkit-daemon0-21swapper/312:21:473
162399780,5rtkit-daemon2197014-21PK-Backend11:19:073
162399780,5rtkit-daemon0-21swapper/308:42:143
162399770,5rtkit-daemon0-21swapper/311:46:393
162399770,5rtkit-daemon0-21swapper/311:46:393
8264899760,72cyclictest1622-21rtkit-daemon10:59:490
162399750,8rtkit-daemon1753-21gnome-shell11:52:123
162399750,6rtkit-daemon2197014-21PK-Backend12:39:143
162399750,5rtkit-daemon0-21swapper/312:09:433
162399740,5rtkit-daemon0-21swapper/311:21:403
162399720,6rtkit-daemon2197014-21PK-Backend11:33:553
162399720,6rtkit-daemon0-21swapper/307:35:433
162399720,5rtkit-daemon0-21swapper/311:07:473
162399720,5rtkit-daemon0-21swapper/308:18:213
162399720,5rtkit-daemon0-21swapper/308:06:433
162399720,5rtkit-daemon0-21swapper/307:22:303
8102426844,16sleep10-21swapper/107:05:031
162399680,5rtkit-daemon0-21swapper/308:37:123
162399670,5rtkit-daemon0-21swapper/308:20:433
82653996626,35cyclictest1753-21gnome-shell09:20:172
82653996626,35cyclictest1753-21gnome-shell09:20:162
82658996520,12cyclictest133049-21cut09:30:253
1761152620,7sleep22197014-21PK-Backend11:35:012
82648996114,29cyclictest126767-21cron09:15:010
82658996023,7cyclictest1069-21runrttasks10:28:123
82658996020,13cyclictest1-21systemd09:45:003
82658996017,14cyclictest2962348-21rs:main3
82658996017,14cyclictest2962348-21rs:main3
8265399602,10cyclictest1753-21gnome-shell07:45:182
162399600,9rtkit-daemon1622-21rtkit-daemon10:42:280
8265399592,10cyclictest1753-21gnome-shell11:59:112
82658995818,11cyclictest186761-21wc12:05:013
162399580,9rtkit-daemon1622-21rtkit-daemon10:45:391
82658995722,12cyclictest42-21ksoftirqd/309:55:053
8265899572,16cyclictest1753-21gnome-shell12:15:113
82658995716,14cyclictest94100-21awk07:40:093
82653995716,18cyclictest1309502basename09:25:202
162399570,9rtkit-daemon1622-21rtkit-daemon11:01:531
162399570,8rtkit-daemon1622-21rtkit-daemon10:39:401
162399570,8rtkit-daemon1622-21rtkit-daemon09:14:351
82658995620,9cyclictest1753-21gnome-shell07:55:113
82658995616,11cyclictest84442-21date07:15:023
82658995521,11cyclictest1753-21gnome-shell10:21:293
82658995519,11cyclictest124546-21grep09:05:333
8265399551,11cyclictest133668-21taskset09:30:402
8265899542,8cyclictest1753-21gnome-shell10:04:423
8265899542,26cyclictest1753-21gnome-shell09:26:043
8265899542,24cyclictest908-21gdbus11:43:233
8265899542,24cyclictest908-21gdbus11:43:223
162399540,8rtkit-daemon1622-21rtkit-daemon07:43:351
8265899539,20cyclictest391irq_work/307:53:333
82658995322,8cyclictest1753-21gnome-shell07:32:443
8265899532,26cyclictest1753-21gnome-shell10:31:523
8265899532,26cyclictest1753-21gnome-shell08:56:143
8265899532,25cyclictest1753-21gnome-shell09:45:073
8265899532,25cyclictest1753-21gnome-shell08:02:403
82653995316,32cyclictest180173-21grep11:45:162
82653995316,32cyclictest180173-21grep11:45:162
162399530,8rtkit-daemon1622-21rtkit-daemon10:22:120
162399530,8rtkit-daemon1622-21rtkit-daemon09:34:121
82658995222,7cyclictest1753-21gnome-shell08:33:453
8265899522,25cyclictest1753-21gnome-shell08:13:443
8265899522,25cyclictest1753-21gnome-shell07:19:433
82658995220,10cyclictest42-21ksoftirqd/312:25:223
82658995219,9cyclictest134268-21sshd09:35:103
82653995220,27cyclictest99194-21cut07:55:022
82653995215,32cyclictest1824-21JS14:31:442
82653995214,33cyclictest156772-21sed10:40:012
8265899511,24cyclictest96931-21/usr/sbin/munin07:45:313
8265399512,7cyclictest1753-21gnome-shell08:19:382
8265399511,46cyclictest0-21swapper/212:32:252
82653995111,35cyclictest438-20systemd-journal10:14:592
82653995110,36cyclictest1753-21gnome-shell10:55:312
162399510,8rtkit-daemon1622-21rtkit-daemon07:45:551
1403732510,9sleep21753-21gnome-shell09:50:322
8265899502,26cyclictest1753-21gnome-shell09:17:223
8265899502,24cyclictest157588-21ntp_states10:40:313
82658995015,12cyclictest42-21ksoftirqd/310:55:003
8265399509,37cyclictest1753-21gnome-shell12:39:302
8265399508,31cyclictest1753-21gnome-shell08:25:352
82653995021,24cyclictest137482-21/usr/sbin/munin09:45:022
82653995019,26cyclictest122602-21ntp_states09:00:272
82653995014,31cyclictest172541-21cat11:25:172
82653995014,31cyclictest151894-21expr10:25:162
82653995010,36cyclictest1753-21gnome-shell10:23:282
8265899491,25cyclictest1753-21gnome-shell10:37:263
8265399498,36cyclictest1753-21gnome-shell10:05:362
8265399491,9cyclictest127472-21sort09:15:222
82653994919,24cyclictest141863-21cat09:55:232
8265399491,43cyclictest1753-21gnome-shell12:18:192
82653994912,32cyclictest189149-21seq12:10:212
82653994912,32cyclictest189149-21seq12:10:212
82653994911,33cyclictest192160-21dpkg12:20:142
82653994911,33cyclictest154405-21cat10:30:322
8265399486,38cyclictest158283-21sh10:45:002
82653994814,30cyclictest1753-21gnome-shell10:54:382
82653994813,29cyclictest164574-21processes11:00:282
82653994812,31cyclictest178465-21grep11:40:232
82653994812,31cyclictest178465-21grep11:40:232
82653994812,31cyclictest168175-21cat11:10:312
82653994810,34cyclictest1753-21gnome-shell07:11:212
1782102480,5sleep1251rcuc/111:40:151
1782102480,5sleep1251rcuc/111:40:151
162399480,6rtkit-daemon1622-21rtkit-daemon09:29:221
162399480,37rtkit-daemon1753-21gnome-shell12:30:253
8265399479,34cyclictest1753-21gnome-shell11:38:542
8265399472,8cyclictest1753-21gnome-shell12:28:552
82653994715,27cyclictest172388-21cat11:25:032
82653994710,32cyclictest1753-21gnome-shell07:19:102
82650994716,8cyclictest1419342netstat09:55:231
162399470,36rtkit-daemon1753-21gnome-shell09:51:393
8265399469,33cyclictest1753-21gnome-shell08:10:332
8265399462,8cyclictest1753-21gnome-shell08:51:442
8265399462,7cyclictest1753-21gnome-shell10:00:312
82653994620,22cyclictest90114-21sshd07:27:432
82653994612,29cyclictest113767-21load08:35:242
82658994520,20cyclictest1753-21gnome-shell09:10:193
8265399457,33cyclictest1753-21gnome-shell11:15:102
8265399457,28cyclictest1753-21gnome-shell07:42:052
8265399452,9cyclictest1753-21gnome-shell09:49:182
82653994512,29cyclictest125864-21latency09:10:232
82653994510,31cyclictest1753-21gnome-shell10:47:222
82653994510,30cyclictest116704-21cpu08:45:142
82650994514,11cyclictest184955-21perf12:00:001
8265399442,38cyclictest1753-21gnome-shell08:02:512
82653994411,28cyclictest438-20systemd-journal08:10:002
82653994410,30cyclictest108766-21proc_pri08:20:262
82650994422,18cyclictest186674-21cron12:05:001
8265399439,30cyclictest148549-21/usr/sbin/munin10:15:202
891882420,10sleep126-21ksoftirqd/107:25:171
82650994219,19cyclictest174435-40apt-check11:30:251
82648994224,13cyclictest83224-21df_inode07:10:190
8264899421,36cyclictest119840-21turbostat08:55:000
1845202420,7sleep32197014-21PK-Backend11:55:323
8265899412,33cyclictest1753-21gnome-shell09:00:183
82650994127,10cyclictest153087-21perf10:30:001
82650994117,12cyclictest855-21dbus-daemon07:25:011
82648994129,8cyclictest1069-21runrttasks07:55:360
162399410,29rtkit-daemon2197014-21PK-Backend08:28:003
82650994023,12cyclictest198836-21/usr/sbin/munin12:40:021
8264899403,33cyclictest105972-21cron08:15:000
82648994028,8cyclictest179444-21turbostat11:45:000
82648994028,8cyclictest179444-21turbostat11:45:000
8264899402,7cyclictest1753-21gnome-shell07:52:250
82648994024,12cyclictest1-21systemd09:20:000
82648994024,11cyclictest109841-21grep08:25:170
82648994023,5cyclictest13-21ksoftirqd/011:37:090
82648994023,13cyclictest179953-40apt-check11:45:180
82648994023,13cyclictest179953-40apt-check11:45:170
82648994022,13cyclictest438-20systemd-journal09:45:420
1290542400,7chrt2197014-21PK-Backend09:20:153
1290542400,7chrt2197014-21PK-Backend09:20:153
1135222400,5sleep00-21swapper/008:35:160
1121932400,7chrt2197014-21PK-Backend08:30:250
8265099395,30cyclictest1069-21runrttasks08:38:471
8265099392,33cyclictest1753-21gnome-shell08:22:561
82650993922,13cyclictest116104-21sshd08:41:431
8264899393,10cyclictest13-21ksoftirqd/009:28:420
8264899392,9cyclictest0-21swapper/010:18:460
82648993927,8cyclictest1753-21gnome-shell08:45:240
82648993923,11cyclictest131987-21apt-get09:30:110
8264899391,8cyclictest2197014-21PK-Backend10:49:530
8264899391,33cyclictest92059-21turbostat07:35:000
8264899391,33cyclictest173907-21turbostat11:30:000
8265899389,6cyclictest1185772cpuspeed_turbos08:50:163
8265099382,32cyclictest1753-21gnome-shell11:39:191
8265099382,32cyclictest1753-21gnome-shell08:05:071
82650993820,13cyclictest438-20systemd-journal11:55:001
82650993819,15cyclictest3242253-21nautilus11:10:011
82650993819,14cyclictest438-20systemd-journal10:55:001
82650993815,18cyclictest124522-21cat09:05:311
82648993826,8cyclictest195793-21tr12:30:180
82648993825,8cyclictest1753-21gnome-shell08:22:590
8264899382,31cyclictest1753-21gnome-shell11:32:390
162399380,26rtkit-daemon0-21swapper/308:47:083
82658993721,11cyclictest1827-21JS14:31:443
82650993721,12cyclictest144868-21grep10:05:161
82650993720,12cyclictest2179-21gvfsd-metadata11:45:031
82650993720,12cyclictest2179-21gvfsd-metadata11:45:031
8264899372,6cyclictest0-21swapper/008:06:370
82648993724,9cyclictest1826-21JS14:31:440
82648993722,11cyclictest120878-21open_files08:55:290
8264899371,31cyclictest123560-21/usr/sbin/munin09:05:250
82658993620,12cyclictest177400-21sshd11:36:433
82658993610,15cyclictest1753-21gnome-shell10:06:253
82650993622,10cyclictest1753-21gnome-shell09:48:291
82650993622,10cyclictest107726-21awk08:20:011
8265099362,19cyclictest140080-21memory09:50:221
8265099362,17cyclictest120863-21ntp_states08:55:291
82650993621,11cyclictest158063-21seq10:42:141
82650993620,12cyclictest189471-21pluginstate12:10:291
82650993620,12cyclictest189471-21pluginstate12:10:291
82650993619,13cyclictest150217-21cat10:20:151
82650993615,6cyclictest141rcu_preempt12:20:151
82650993614,8cyclictest26-21ksoftirqd/109:37:031
82648993625,7cyclictest181732-21acpi11:50:140
8264899362,12cyclictest1358-1kworker/u9:108:40:160
8264899362,10cyclictest122253-21hddtemp_smartct09:00:200
82648993618,13cyclictest146488-21cat10:10:150
82648993610,12cyclictest438-20systemd-journal10:00:010
1966242360,6sleep12197014-21PK-Backend12:30:351
1664032360,14chrt13-21ksoftirqd/011:05:340
162399360,21rtkit-daemon1753-21gnome-shell09:40:371
82658993522,9cyclictest1753-21gnome-shell10:47:593
8265099353,29cyclictest1069-21runrttasks10:00:041
82650993524,7cyclictest170841-21tr11:20:151
82650993520,12cyclictest0-21swapper/107:10:241
82650993520,11cyclictest918-21gdbus07:51:561
82650993520,11cyclictest119313-21cat08:50:311
82650993518,13cyclictest85235-21cpuspeed_turbos07:15:181
82648993524,7cyclictest197524-21tr12:35:180
8264899352,30cyclictest0-21swapper/012:23:030
8264899351,9cyclictest2197014-21PK-Backend07:43:250
82648993519,12cyclictest3242255-21gdbus10:55:010
82648993519,12cyclictest3242255-21gdbus09:25:010
82648993519,12cyclictest3242255-21gdbus09:25:010
82648993519,12cyclictest153514-21cut10:30:140
82648993519,11cyclictest152414-21which10:25:280
8264899351,11cyclictest172132-21chrt11:24:490
8264899351,10cyclictest2197014-21PK-Backend12:10:330
8264899351,10cyclictest2197014-21PK-Backend12:10:320
162399350,21rtkit-daemon1622-21rtkit-daemon09:23:421
162399350,21rtkit-daemon1622-21rtkit-daemon09:23:421
8265899341,9cyclictest2197014-21PK-Backend10:58:353
82658993418,12cyclictest149050-21awk10:15:333
82650993422,9cyclictest1753-21gnome-shell11:12:501
82650993419,11cyclictest99313-21dpkg07:55:111
82650993419,11cyclictest173905-21apt-get11:30:011
82650993419,11cyclictest122177-21expr09:00:161
82650993419,11cyclictest111166-21sh08:29:591
82650993417,13cyclictest1753-21gnome-shell09:16:401
8264899342,5cyclictest0-21swapper/007:22:320
82648993423,7cyclictest96012-21tr07:45:140
82648993423,7cyclictest183488-21apt11:55:120
82648993421,9cyclictest13-21ksoftirqd/008:01:260
82648993420,10cyclictest186685-21cron12:05:000
82648993419,10cyclictest90314-21/usr/sbin/munin07:30:020
82648993416,14cyclictest2181-21gdbus09:55:010
8264899341,15cyclictest2197014-21PK-Backend11:01:160
82650993323,7cyclictest146543-21cpuspeed_turbos10:10:161
82650993322,7cyclictest194879-21timerandwakeup12:25:311
8265099332,27cyclictest154855-21munin-run10:35:011
82650993319,10cyclictest190733-21expr12:15:171
82650993318,11cyclictest118163-21cut08:50:011
8264899332,27cyclictest1753-21gnome-shell09:38:340
82648993319,10cyclictest13-21ksoftirqd/012:15:250
82648993318,11cyclictest93833-21cron07:40:000
82648993318,11cyclictest106747-21sed08:15:240
82648993317,12cyclictest144011-21grep10:00:320
82648993317,11cyclictest155295-21cstates10:35:160
82648993317,11cyclictest145067-21sh10:05:170
8264899331,13cyclictest2197014-21PK-Backend09:42:060
1693202330,7chrt2197014-21PK-Backend11:15:150
8265899322,8cyclictest1753-21gnome-shell10:14:503
82650993224,5cyclictest0-21swapper/107:34:091
8265099322,25cyclictest105257-21ntp_states08:10:281
82650993221,6cyclictest26-21ksoftirqd/112:09:341
8265099322,13cyclictest162391-21df10:55:151
82650993218,10cyclictest93132-21cut07:35:291
82650993218,10cyclictest169078-21wc11:15:141
82650993217,11cyclictest149608-21cron10:20:001
82650993217,11cyclictest102527-21basename08:04:591
82648993210,18cyclictest1753-21gnome-shell11:13:440
82648993123,5cyclictest2197014-21PK-Backend07:15:580
82650993018,8cyclictest112126-21cut08:30:231
8264899308,18cyclictest1753-21gnome-shell12:08:560
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional